Do Performance Aspirations Matter for Guiding Software Configuration Tuning? An Empirical Investigation under Dual Performance Objectives

Configurable software systems can be tuned for better performance. Leveraging on some Pareto optimizers, recent work has shifted from tuning a single, time-related performance objective to two intrinsically different objectives that assess distinct aspects of the system, each with varying aspirations satisfied, e.g., “ latency is less than 10s ” while memory usage no more 1GB ”. Before we design crucial engineering decision make therein how handle requirements clear in process. For this, community takes alternative optimization models: either quantifying and incorporating into search guide tuning, or not considering during but purely using them later decision-making process only. However, despite being determines an optimizer designed tailored, there rather limited understanding which model should chosen under what particular circumstance, why. In this article, seek close gap. Firstly, do through review over 426 articles literature 14 real-world datasets, summarize four requirement patterns quantify configuration tuning. Drawing these, then conduct comprehensive empirical study covers 15 combinations state-of-the-art patterns, types aspiration space, three eight systems/environments, leading 1,296 cases investigation. Our findings reveal (1) realism key factor whether they used tuning; (2) given position realistic landscape are important choice, matter extents improvement; (3) available budget also influence choice unrealistic it insignificant ones. To promote open science practice, our code dataset publicly at: https://github.com/ideas-labo/aspiration-study .